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WHAT IS CLAIMED IS: 

A method for integrated retrieval and annotation of stored 
images involving a computer application in which text is input by a user, said 
method comprising the steps of: 

a) providing metadata associated with the stored images; 

b) running a user application in which text is entered by a user; 

c) continuously monitoring text typed by the user into the 
application to isolate the context expressed by the text; 

d) matching the context with the metadata associated with the 
stored images, thereby providing one or more matched images; 

e) retrieving and displaying the matched images; and 

f) utilizing the context to provide suggested annotations to the user 
for the matched images, together with the capability of selecting certain of the 
suggested annotations for subsequent association with the matched images. 

2. The method as claimed in claim 1 further comprising the 
step of providing the user with the capability of inserting selected ones of the 
matched images into the text of the application. 



3. The method as claimed in claim 1 wherein the stored 
images are stored in a data base and step f) further includes automatically 
updating the metadata for the matched image stored in the data base with the 
suggested annotations. 

4. The method as claimed in claim 1 wherein the metadata 
includes keywords related to the images and the context likewise comprises 
keywords. 



5. The method as claimed in claim 1 wherein the metadata 
includes content data related to the images and the context likewise comprises 
content data. * 
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The method as claimed in claim 1 wherein the metadata 



includes temporal data related to the images and the context likewise comprises 
temporal data. 



entering text at a particular cursor location and the context is determined based 
upon the proximity of typed words to the cursor location. 



step of providing the user with the capability of adding other annotations to the 
matched images. 



comprises ranking the matched images according to context, thereby providing 
ranked images that may be displayed in step e) according to their rank. 

10. The method as claimed in claim 9 wherein the context 
comprises keywords and the ranking is based on the number of matched keywords 
for each matched image. 

1 1 . The method as claimed in claim 9 wherein the context and 
metadata include temporal data and the ranking is based on the temporal distance 
between the context and the metadata associated with each matched image. 

12. The method as claimed in claim 9 wherein the context 
comprises keywords and temporal data and the ranking is based on the 
intersection of their occurrence for each matched image. 



A method for utilizing images that are stored in a data base 
with an e-mail application in which text is input by a user into a message, said 
method comprising the steps of: 

a) providing metadata associated with the stored images; 



7. 



The method as claimed in claim 1 wherein the user is 



8. 



The method as claimed in claim 1 further comprising the 



The method as claimed in claim 1 wherein step d) further 
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b) running an e-mail application in which text is entered by a user 
into a message window; 

c) continuously monitoring text typed by the user into the message 
window to isolate keywords in the text; 

d) matching and ranking the keywords with the metadata 
associated with the stored images, thereby providing one or more matched images 
that are ranked according to the keywords; 

e) retrieving and displaying the matched images alongside the 
message window in order of their rank; and 

f) providing the user with the capability of moving selected ones of 
the matched images into the message window. 

14. The method as claimed in claim 13 further comprising the 
step of utilizing the context to automatically provide new keyword annotations for 
the matched images. 

15. The method as claimed in claim 13 further comprising the 
step of utilizing the context to provide new keyword annotations for the matched 
images, together with the capability of manually selecting certain of the 
annotations for subsequent association with the matched images in the data base. 

16. The method as claimed in claim 14 wherein the step of 
utilizing the context further includes automatically updating the metadata for the 
matched image with the annotations. 

17. The method as claimed in claim 15 wherein the step of 
utilizing the context further includes automatically updating the metadata for the 
matched image with the annotations. 

18. The method as claimed in claim 13 wherein the ranking is 
based on the number of matched keywords for each matched image. 
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19. The method as claimed in claim 13 wherein the keywords 
and metadata include temporal references and the ranking is based on the 
temporal distance between the keyword and the metadata associated with each 
matched image. 

20. The method as claimed in claim 13 further comprising the 
step of utilizing the context to either automatically or manually select new 
keyword annotations for the matched images, wherein the ranking is based on a 
higher confidence level for the manually selected keyword annotations. 

21 . The method as claimed in claim 13 wherein the keywords 
and metadata include spatial or geographic references and the ranking is based on. 
the spatial or geographic correspondence between the keyword and the metadata 
associated with each matched image. 

An autonomous agent for use in automatically interfacing a 
text-based application with a picture archive containing representations of images 
and metadata associated with the images, said agent comprising: 

a language analyzer for continuously monitoring text entered by 
the user into the application and extracting keywords appropriate to the context 
surrounding the entered text; 

a retrieval agent for retrieving images from the archive on the basis 
of one or more matches between the extracted keywords and the metadata; 

an annotation agent for providing suggested annotations to the user 
for the matched images based on the extracted keywords, together with the 
capability of selecting certain of the suggested annotations for subsequent 
association with the matched images; and 

a picture database viewer for displaying the retrieved images to the 
user and for enabling the images to be incorporated into the application, as desired 
by the user. 
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23. The agent as claimed in claim 22 wherein the retrieval 
agent ranks the retrieved images based on attributes of the keywords, and the 
picture database viewer displays the retrieved images in order of their ranking. 

24. The agent as claimed in claim 22 wherein the application is 
an e-mail application. 

25. The agent as claimed in claim 22 wherein the picture 
database viewer incorporates a retrieved image into the application by a drag and 
drop operation. 

26. The agent as claimed in claim 22 wherein the picture 
database viewer incorporates a retrieved image into the application by a one click 
selection of the image. 

27. The agent as claimed in claim 22 further comprising a 
manual annotation agent for adding annotations to the retrieved image and for 
storing the added annotations to the metadata stored in the archive. 



agent retrieves images from the archive on the basis of matching between the 
extracted keywords and temporal metadata identifying the time and date when the 
images were captured. 

29. The agent as claimed in claim 22 further comprising an 
automatic import agent for automatically entering images into the archive from 
one or more specified image sources. 



28. 



The agent as claimed in claim 22 wherein the retrieval 



30. The agent as claimed in claim 29 wherein the retrieval 
agent automatically retrieves the entered images and the picture database viewer 
automatically displays the entered images. 



